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Abstract 


We investigated how high school students taking a university preparatory economics 
course would engage with the learning and assessment components of a Behavioural 
Economics MOOC that was integrated into their school-based course. Students were 
divided into two groups, MOOC-only, with no teacher support, and blended-mode, with 
weekly tutorials. MOOC only students scored slightly lower on a teacher designed 
knowledge test but scored slightly higher in a MOOC test. Although the MOOC-only 
students watched more unique videos, the blended-mode students stayed more on-track 
with the MOOC. The blended-mode students showed more persistence in retaking 
quizzes, yet they scored lower than the MOOC-only students. 
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Introduction 


Massive online open courses (MOOCs) have been used in various ways often in the form 
of a flipped classroom, to complement traditional classroom teaching through 
integration of a whole course or specific parts of a course both in K-12 and in higher 
education. High school students, for example, could use MOOCs for university 
preparation in the absence of available face to face or online courses or in addition to 
them (Vihavainen, Luukkainen, & Kurhila, 2013). Furthermore, contents of a live or an 
archived MOOC could be integrated into an existing course in a hybrid format (Bruff, 
Fisher, McEwen, & Smith, 2013). 

In this research we investigated how high school students taking a university 
preparatory economics course would engage with learning and assessment components 
of a Behavioural Economics MOOC that was integrated into their school-based course. 
Of specific importance to this research was probing any potential difference in students’ 
use of various components of the integrated MOOC depending on receiving 
supplementary instructional support or studying on their own. For this purpose, three 
weeks of the "Behavioural Economics in Action" (BEioix) MOOC were chosen for 
integration into the school-based course "Analyzing Current Economic Issues" (ACEI). 
Twenty nine students from a Canadian independent university preparatory school who 
had enrolled in ACEI were randomly assigned to two groups for the BEioix integration 
period with one group studying on their own and the other receiving weekly tutorial 
sessions from their classroom teacher. Here, we report on how students in the two 
groups engaged with the learning and assessment components of BEioix and persisted 
throughout the three weeks of integration. Also, we compare students’ outcomes based 
on pre-post BEioix-integration content knowledge surveys and BEioix assessments. 


Review of Related Literature 


Research on online university preparatory courses, MOOC-like university preparatory 
courses, and MOOC integration in credit courses informed our study. 

Online University Preparatory Courses 

Online advanced placement (AP) courses could increase students' access when face to 
face options are not viable due to restricted budgets or geographical location. In 
Newfoundland and Labrador, for example, the percentage of students in rural schools 
who enrolled in advanced-level courses through the Centre for Distance Learning and 
Innovation was higher than the provincial average (Barbour, & Mulcahy, 2013). 
Similarly, students in Florida who enrolled in online AP courses available to all 
students, regardless of income level and district budget, scored higher than the state 
average in their AP exams (Johnston, & Barbour, 2013). Yet students who participated 
in a follow up interview reported a preference for face to face classes where they could 
readily communicate with a teacher and improve their performance based on teacher 
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feedback. Teacher presence in online secondary school courses positively affected 
student outcomes in introductory mathematics courses (Liu & Cavanaugh, 2012). For 
more advanced courses, on the other hand, the amount of teacher feedback was 
negatively related to course outcomes which may indicate that to be successful in more 
advanced online courses, high school students need to be self-directed and less reliant 
on teacher feedback. Time spent in the online course environment was another factor 
that would have a positive effect on students’ assessment outcome. 

MOOCs for Curriculum Enrichment and Reach-Ahead 

Some MOOCs have imposed minimal prerequisites for registration, and high school 
students have already established themselves as a demographic group in MOOCs of 
various difficulty level (Breslow et al., 2013). 2.39% of survey respondents of an 
electronics MOOC, with suggested academic prerequisite of having advanced knowledge 
of electricity and magnetism, reported their highest education level to be below high 
school diploma (DeBoer, Stump, Seaton, & Breslow, 2013). In terms of course 
achievement, however, total points gained by those students with junior high school 
education level were comparable with students holding a bachelors degree. 

Aside from MOOCs offered to the general public, specialized MOOCs have been 
developed for middle and high school students, either as university preparatory courses 
or as high school level courses. To address the paucity of pedagogically sound high 
school level computer science curricula, a mini course titled “Foundations for Advancing 
Computational Thinking” (FACT) was developed on the Stanford OpenEdX platform 
and piloted in a public middle school (Grover, Pear, & Cooper 2014). The FACT mini 
course has demonstrated promising results regarding students’ learning and it is being 
offered as a MOOC for either self-study or for teachers to use in their computer science 
classrooms. A similar project, MyCS, geared towards middle school students, was also 
considered for further development as a teacher-mediated computer science MOOC 
(Schofield, Erlinger, & Dodds, 2014). 

To reinforce students’ knowledge of science and mathematics and to facilitate their 
entrance to universities, an Italian school board implemented high school specific 
online open courses by recording classroom lectures and making them available online 
(Cannesa & Pisani, 2013). Test results showed that students who watched online lecture 
videos in addition to their face to face classroom scored higher than those who only 
relied on face to face classroom learning. Although not a typical MOOC in terms of 
design, this project supports the notion that self-paced open access material, geared 
towards high school students may increase students’ depth of learning and academic 
success. In another study, a university preparatory mathematics MOOC was developed 
to increase mathematical proficiency of incoming students (Daza, Makriyannis, & 
Rovira Riera, 2013). First year university students were encouraged to participate in the 
MOOC voluntarily while the MOOC was also open to the public. Initial findings, showed 
students level of satisfaction with the contents of the MOOC, although the impact of 
MOOC learning on students’ success was not measured due to the scope of the study. 
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While several projects such as the University of Wisconsin’s College Readiness Math 
MOOC and the MIT physics MOOCs offer preparatory courses to secondary school 
students, research on the impact and implications of such initiatives is still in its 
infancy. 

MOOC Integration 

With respect to integrating existing MOOCs into regular classroom instruction, flipped 
classrooms could offer one model. In flipped classrooms, the online component, for 
example the MOOC, allows students to prepare for in-class discussion by reviewing 
relevant material beforehand and at their own pace. In other models of integration, even 
MOOC assessments could count towards students’ evaluation of the on-campus course. 
The instructor of a machine learning graduate course, for example, integrated an 
archived MOOC into his face to face course where students were required to take MOOC 
quizzes and write MOOC assignments, submitted to their instructor, in addition to 
covering learning components of the MOOC. Students responded positively to the 
experience and noted that the face to face classes would help them keep track of the self- 
paced MOOC component (Bruff et al., 2013). 

Research Questions 

In this study, we examined potential impacts of the presence or absence of teacher 
support on secondary school students’ behavior and outcome during a MOOC- 
integration initiative. Three research questions guided our study: 

• How do students in MOOC-only and blended-mode groups differ in their 
learning outcomes? 

• How do students in MOOC-only and blended-mode groups differ in the level of 
engagement with the learning and assessment components of the integrated 
MOOC? 

• How do students in MOOC-only and blended-mode groups differ in their 
persistence to complete the activities of the integrated MOOC? 


Methods 


A case study encapsulates our research design in examining the impact of different 
levels of instructional support on students’ engagement with learning and assessment 
components of BEioix and their outcome as evident in MOOC quizzes and knowledge- 
based tests. To this end, BEioix integration in the ACEI course could be an 
instrumental case (Stake, 1995) that allows us to gain insight into the issue of using 
MOOCs in the context of a secondary school course. 
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Research Context 

This study was conducted in the context of a university preparatory economics course at 
a competitive independent Canadian school from October 14 2013 to November 21 
2013. The two participating class sections were taught by an experienced teacher, also a 
co-investigator of the study. The age range of students enrolled in this course was 
between 15 to 17 years. Twenty nine students agreed to participate in the study out of a 
total of 32 students. Participation in this study was voluntary and had no impact on 
students’ formal evaluation and course outcomes. 

BEioix, a six week MOOC on edX, was first offered in October 2013. Consequently, this 
study coincided with the live offering of this MOOC. Below we discuss curriculum 
design considerations, research timeline, participant recruitment, and grouping 
procedures. 

Curriculum Design 

Integration of BEioix in ACEI was temporally bound between October 14, 2013 and 
November 18, 2013, weeks 1 to 5 of the BEioix MOOC. Before the study started, the 
teacher compared the contents and objectives of BEioix and ACEI to find areas of 
overlap where BEioix could be most effectively integrated. 

Weeks 1, 2, and 5 of BEioix MOOCs were selected for integration. ACETs start time 
coincided with the first week of BEioix. Timeline, themes, learning components and 
assessment components of weeks 1, 2, and 5 of BEioix were as follows. Note that only 
those learning components that preceded a quiz or a debate were accounted for. 

• Week 1 (October 14 to 20): Introduction to Behavioral Economics. 

o 5 lecture videos, one article accompanied by slides, and 6 quizzes 
o Debate video and week 1 debate 

• Week 2 (October 21 to 27): Mental accounting 

o 7 lecture videos and 7 quizzes 
o Debate video and week 2 debate 

• Week 5 (November 11 to 17): Nudging 

o 8 lecture videos, 2 articles, 10 quizzes 
o Debate video and week 2 debate 
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Prior to the start of the study, students were randomly divided into two groups for 
research purposes: a MOOC-only group with 14 students and a blended-mode group 
with 15 students. 

In the three weeks outlined above, regular classes were not held. Instead, students in the 
MOOC-only group would study BEioix independently with no support from their 
teacher. On the other hand the blended-mode group students met their teacher once a 
week in an hour-long tutorial session. Materials from these sessions were given to 
MOOC-only students after week 5. 

Tutorials were structured to include three common sections: Questions and answers, 
Experiment/Business problem, and an Individual exercise. The purpose of these tutorial 
sessions was to elaborate on BEioix themes and engage students in applying what they 
had learned during that week. The dates of these tutorials were: week 1, October 21, 
2013; week 2 , October 28, 2013; and week 5 , November 18, 2013. 

Students would attend regular classes in weeks 3 and 4, between October 29 and 
November 10, 2013. Upon the culmination of week 5 of BEioix, regular classes were 
resumed for the rest of the school term. 

Students received an email prior to the start of their course that explained the study and 
its goal. This email was followed by another orientation email sent from the teacher on 
the first day of BEioix asking students to sign up for the MOOC on edX and also 
recommending that they refrain from starting the course until October 16th where a face 
to face orientation class meeting would be held. 

Time and date for knowledge-based pre and post tests were also shared with the 
students. On October 16, 2013 the teacher met all of the students where the students 
received further explanation about BEioix and the study and took a knowledge-base 
pre-test. The teacher explained that the students would not be evaluated on their 
participation in BEioix and that participation and the extent of engagement were 
absolutely the responsibility and choice of the students. 

Data Sources 

To maintain student anonymity, we refrained from using any data that could lead to 
revealing their identity including their contributions to BEioix discussion forums and 
weekly debates. We collected data from the following sources. 

• Clickstream data of BEioix between October 14, 2013 and November 21, 2013. 
We selected specific events in the clickstream that could reveal differences in 
use of and engagement with the integrated BEioix resources between the 
MOOC-only and the blended-model groups. For videos, we specifically 
considered "play_video" and "pause_video" events to determine if a video was 
watched. M Load_video M event were disregarded as it is generated when a page 
containing a video is loaded and this does not necessarily mean that the video 
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was played. In the duration of BEioix integration students couldwatch a 
maximum of 6 debate and debate debrief videos and 20 lecture videos. 

• Content knowledge pre-post questionnaire. The teacher designed a 10-item 
true/false questionnaire that students wrote in class in the week of October 14, 
2013 and again in the week of November 18, 2013 after the MOOC integration 
culminated. 

• BEioix quizzes in weeks 1, 2, and 5 and BEioix test in week 3. The test in week 
3 covered material from weeks 1 and 2. Data related to quizzes taken, number of 
attempts, and final score was extracted from the database for students in both 
groups. Another test was written in week 6 of the course but students’ 
performance in this test could not be extracted from the raw data. 

Data Analysis 

Three dimensions of student engagement, student persistence, and student outcomes 
during the duration of their MOOC participation were of importance to this research. 

Student engagement related to students’ use of and access to BEioix learning 
components, specifically lecture videos that preceded each quiz, debate videos, and 
debate debrief videos. These learning components were located within the clickstream 
data by their unique identifiers. Four quizzes were preceded directly by either articles or 
slides. We could not track them in the clickstream data and thus did not include them in 
our analyses. For student engagement dimension, we compared the two groups on 
unique videos accessed, total number of videos accessed, and videos accessed on-track 
with BEioix pace. 

Student persistence referred to the number of quizzes they took in weeks 1, 2, and 5 of 
the integration and if quizzes were retaken until a perfect score was achieved. The last 
dimension, student outcome, attended to differences between MOOC-only and blended- 
mode groups on their performance in BE101X test, overall quiz score, and in content 
knowledge pre-post tests. We would like to note that technical problems in extracting 
clickstream data largely affected the depth and scope of our analysis. 


Findings 


Students’ Learning Gains and Outcome 

Students’ scores in content knowledge pretest, written at the start of BEioix 
integration, and posttest, written after the integration was over, showed an increase in 
their knowledge of Behavioural Economics. As illustrated in Figure 1, MOOC-only 
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students scored lower in the pretest (M= 4.91; SD= 1.08) compared to blended-mode 
students (M= 6.06; SD= 1.86). In the posttest, despite performing better than the 
pretest (M= 7.75; SD= 1.42), MOOC-only students still gained lower scores than the 
blended-mode students (M= 8.53; SD= 1.68). A two sample t-test did not show a 
significant difference between the two groups on their posttest scores; f(25)=i.28, 
p=o.i. 

A BEioix test written in week 3 provided another measure for students’ outcome. 
Unlike the content knowledge posttest results, students in the MOOC-only group scored 
higher (M= 12.73, SD=1.38) than the blended-mode group students (M=n.53, 
SD=3 .o 6). A two sample t-test did not reveal a significant difference between the two 
groups, t(23)^0.94; p=o.i7. 


We investigated if the observed overall difference between the MOOC-only and the 
blended-mode groups in terms of their pretest and posttest scores also reflected in their 
level of persistence and engagement with learning and the assessment components of 
the integrated MOOC. 



□ MOOC-only group 

□ Blended-mode grop 


Figure 1. Students’ performance in pretest and posttest. 


Overall Activity During MOOC Integration 

During the running of the course, students were free to login to BE101X whenever they 
desired. However, only student access to BE101X between October 14, 2013 and 
November 21, three days after the third tutorial session, was specifically relevant to our 
research. We started by, first, examining the number of students in both groups and in 
total who accessed the integrated BE101X at least once in each day between October 14, 
2013 and November 21. Although weeks 3 and 4 of BE101X were not integrated into 
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ACEI, we included these two weeks in our analysis as we assumed that students may 
have reviewed week l and week 2 material or have caught up with activities that they 
had missed. 

The number of students who logged into the integrated BEioix in weeks 1, 2 and 5, in 
addition to weeks 3 and 4 in between, is shown in Figure 2. We corroborated high and 
low levels of activity with important dates during the integration period. Note that any 
activity before October 16 is negligible as students were not yet divided into groups. 

The highest level of activity occurred on the two days when BE101X weeki&2 and weeks 
tests were administered, October 31, 2013 and November 21, 2013 respectively. The day 
of the first tutorial showed another peak of activity with 7 MOOC-only and 9 blended- 
mode students online. A high activity level is observed on November 18 for the blended- 
mode group, which can be attributed to the third tutorial session. As expected, the level 
of activity was low for both groups after the October 31 test and remained relatively low 
throughout week 4. Both groups were comparable in the level of activity until November 
16, 2013 when a peak of activity is evident for the blended-mode students that was 
sustained until November 18 and then dropped quickly in the next two days. In that 
three-day period, students in the MOOC-only group maintained a lower level of activity. 
However, their activity level steadily increased and eventually equated with the blended- 
mode students in the two days leading to test 2 on November 21, 2013. 



- MOOC-only group (n=14) Blended-mode group (n=15) 


Figure 2. General trend in students’ activity in the integrated BE101X and in the two 
weeks in between based on the number unique daily online users. 


Use of Resources: Unique Number of Videos Accessed 

We examined students’ access in both groups to the lecture videos that would precede a 
quiz in weeks 1, 2, and 5. First, we calculated the number of unique lecture videos, 
debate videos, and debate debrief videos students had accessed in the duration of 
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BEioix integration, including weeks 3 and 4. Table 1 summarizes descriptive statistics 
relevant to unique videos that students in both groups watched. 

Table 1 

Students'Access to Lecture Videos and Debate/Debrief Videos 


Group 

Lecture videos 

Debate/Debrief videos 

MOOC-only (n=i4) 

M=i8.00 

SD= 4.33 

M=2 

SD=i.35 

Blended model (n=i5) 

M=i6.46 

SD=5.42 

M=l.8 

SD=i.65 


MOOC-only students on average watched, that is, played, more unique videos than 
students in blended-mode group. Overall engagement level with lecture videos based 
on the average number of lecture videos watched was close to a maximum of 20 videos 
in both groups. 

To better understand any noticeable differences in how students in the two groups 
accessed unique lecture videos of the integrated MOOC, we graphed the percentage of 
students in each group who watched lecture videos of week 1, week 2, and week 5 at 
least once any time during the integration and in the two weeks in between. As depicted 
in Figure 3, 80% of all lecture videos, were watched by more students from the MOOC- 
only group. For the rest of the lecture videos, the difference between the number of 
blended group students and MOOC-only group students who watched those videos was 
a maximum of two students. Throughout the three weeks and between the two groups, 
the percentage of students watching lecture videos fell slightly with only one of 7 videos 
being watched by more than 90% of the students. This number was 6 out of 6 for the 
week 1 video and 3 out of 7 for the week 2 lecture videos. Still, the overall lowest 
percentage of students watching a lecture video, which is a week 5 lecture video, was 
nearly 67% by students in the blended-mode group. 
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100 



□ MOOC-only group 

□ Blended-mode group 


Figure 3. Percentage of students in each group who watched lecture videos. 


Remaining On-Track or Lagging Behind 

Students in both groups watched most of the lecture videos belonging to three weeks of 
BEioix integration. However, we questioned if students would have followed the course 
week by week or would have lagged behind and compensated later. Thus, for both 
groups, we examined if students watched lecture videos of a given week at least once 
during that week. 

Table 2 summarizes the average percentage of lecture videos in weeks 1, 2, and 5 that 
were watched within those specific weeks. As Table 2 shows, on average, students in 
blended-mode group following the BEioix course pace watched lecture video more than 
their peers in the MOOC-only group. Week 2 engaged the highest number of students in 
both groups in watching lecture videos as MOOC-only students and blended-mode 
group students watched 43.88% and 63.81% of lecture videos between November 21, 
2013 and November 27, 2013. 

Table 2 


Percentage of Students who Watched Lecture Videos On-Track 



Week 1 videos 

Week 2 videos 

Week 5 videos 

MOOC-only group (n=i4) 

36.9% 

43.88% 

33.67% 

Blended model group (n=i5) 

46.67% 

63.81% 

51-43% 
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We, then, considered if there were students in both groups who consistently remained 
on-track during the three weeks of integration or consistently lagged behind. In weeks 
l, 2, and 5, four MOOC-only students, 28.57%, completely stayed on track whereas six 
others, 48.85%, completely lagged behind and watched none of the lecture videos 
belonging to those weeks. These numbers for the blended-mode group were three 
students, 20%, and four, 26.66% respectively. 

Figure 4 and figure 5 show the difference between the two groups regarding following 
the pace of the integrated MOOC by watching lecture videos in Weeks 1, 2, and 5 on- 
track, in more detail. More students from blended-mode group followed the pace of the 
course for more than 80% of the lecture videos. Consequently, although a higher 
percentage of MOOC-only students watched all videos following the pace of the course, 
blended-mode students as a group remained on-track more consistently. 



Figure 4. MOOC-only students’ pace of watching lecture videos. 



Figure 5. Blended-mode students’ pace of watching lecture videos. 
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Engagement With and Persistence In Quizzes 

Students could take a total of 29 quizzes with a maximum total quiz points of 29. A quiz 
would only be graded if students chose to submit their answer. As a result, a zero quiz 
point would either mean that the student submitted an incorrect answer, or else, the 
student did not take the quiz at all. To examine students’ performance in quizzes in the 
MOOC-only and blended-mode groups we considered total points received in all of the 
29 quizzes regardless of whether a quiz was actually submitted for grading. We factored 
such differences into our analyses of how students attempted to retake quizzes to 
achieve maximum points possible. 

Students in the MOOC-only group received slightly more points in quizzes (M= 24.95; 
SD=5.98) than students in blended-mode group (M=23.74; SD=743). Regarding the 
number of quizzes taken, out of 29 quizzes, MOOC-only group submitted more quizzes 
for grading (M=26.42; SD=6.o8) compared to blended-mode group students (M= 
24.53; SD=7.69). 

Finally, we investigated students’ persistence while taking quizzes in terms of achieving 
maximum quiz points possible for the number of quizzes taken. Blended-mode group 
students outperformed MOOC-only students in persistence as 40% of the former versus 
21.42% of the latter correctly answered the quizzes that they submitted for grading. 


Discussion 


Of specific importance to this research was probing any potential differences in 
students’ use of learning and assessment components of the integrated MOOC 
depending on receiving supplementary instructional support or studying on their own. 
Students’ scores in the pretest, written before starting the integrated BEioix, and 
posttest, written after the integrated BEioix was over, showed an increase in their 
knowledge of behavioural economics. Thus, the integration had no adverse effect on 
students’ learning. 

With the opportunity to expand on BEioix content in the three tutorial sessions, we 
would expect blended-mode students to watch more video lectures and be more 
engaged with these learning components. Surprisingly, MOOC-only students on average 
watched more unique videos than students in blended mode group. Regarding students’ 
performance in MOOC quizzes, MOOC-only students were at no disadvantage as they 
received more quiz points compared to blended-mode students. However, we noticed 
that students from the blended-mode group were more likely to retake quizzes to 
achieve complete quiz points. 

MOOC quizzes, considered as data sources for this study, may only reveal students’ 
ability to recall information. The specific MOOC discussed in this study also contained 
knowledge transfer and application assignments and weekly debates that were not 
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included as data sources. We also inferred from clickstream data that students 
participating in this study did not participate in the MOOC’s discussion forum. 
However, with the evidence that students would access MOOC resources and take 
quizzes with no direct contact with their classroom teacher, we posit that MOOCs could 
provide another means for reaching ahead and preparing for university for students 
who rely on online advance course for reasons such as geographical location (Barbour & 
Mulcahy, 2013). 

In this study, we questioned if students would have followed the course week by week or 
would have lagged behind and compensated for it before the tests in week 3 and week 6 
to cover lecture videos. Fewer students in the blended-mode group lagged behind, which 
could be attributed to the weekly face to face tutorials. Previous research confirms this 
finding (Bruff et al., 2013) that instructional presence may positively affect students 
following the expected pace of integrated content. 

Although a higher percentage of MOOC-only students watched all videos following the 
pace of the course, blended-mode students as a group remained on-track more 
consistently. Knowing that they would discuss lecture videos of a given week during the 
tutorial session, these students would have preferred to attend the class prepared rather 
than postponing watching lecture videos to a few days before test 1 in week 3 and test 2 
in week 6. 

Empirical literature to investigate potential learning benefits of developing MOOCs or 
MOOC-like initiatives for specific age or grade level group is still in its infancy. Existing 
studies either report on preliminary findings in the form of overall satisfaction (Daza et 
al., 2013) or improved test results (Cannesa & Pisani, 2013) or propose the development 
of a MOOC based on pilot projects (Grover et al., 2014; Schofield et al., 2014). We 
contributed to this growing line of research, by comparing how high school students 
engage with a MOOC integrated into their school-based course, in a self-directed 
manner or with teacher support. 


Conclusion 


The results of our study are promising regarding integrating MOOCs in school-based 
courses in a self-study manner. Students engaged with learning and assessment 
components of an integrated MOOC in a self-study manner. It is important to note 
however, that participants of this study were high achieving and intrinsically motivated 
students which may have affected their level of engagement. Integrating MOOCs into 
school-based courses entails curriculum design challenges as classroom teachers need 
to find relevant content that enriches their existing curriculum. One implication would 
be for teachers to carefully examine the added cognitive value of MOOC integration. 
Another challenge is the persistence of an integrated MOOC over time. In case an 
integrated MOOC is removed from its provider platform, the teacher may have to look 
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for a replacement or abandon the integration. MOOCs specifically developed for school 
courses may alleviate such problems. 

Understanding students’ experience through interviews and detailed reflection notes 
would shed more light on their perception of the usefulness of the integration and the 
challenges they may have faced. This study only included quantitative measures but our 
next round of research will also consider in-depth qualitative data related to the MOOC 
integration design process, student interviews and reflection, and a knowledge 
integration project as data sources so that we are able to compare the intended and the 
implemented MOOC integration effort in more depth. 
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